Repository navigation
feat(short_drama): turn a novel into vertical drama episodes on TRTMC - #1932
Draft
chaofengw-nv wants to merge 69 commits into
Draft
chaofengw-nv wants to merge 69 commits into
chaofengw-nv wants to merge 69 commits into
Conversation
Restore the family-owned original image-reference pipeline and its staged bundle writer on the current API. Keep automatic support discovery and the original single-window public behavior unchanged. Add private synchronized AV guide layout, target-only sampling, clean-audio AdaLN cache invalidation, global window clocks and channel-correct latent-tail accumulation. Add a real backend guide trajectory oracle without changing the existing FP32/BF16 criteria. Validation: 280 CPU family/bundle regressions, one new guide profile test, four C++ host executables, 176 exact upstream layout vectors and 13 exact upstream clock windows passed. The new GPU guide oracle is not yet executed. The retained trained encoder/video failures remain unqualified; this commit does not advertise trained continuation or production quality. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Add a separate 226-frame experimental bundle profile with synchronized video/audio latent guides, two complete 49-forward windows and cumulative decoding. Preserve the original single-window profile and residency options. Reject child profile limits before first-window inference, retain each stereo channel prefix, and serialize the complete request across the AdaLN cache transition. Add an independent native 98-forward oracle with synthetic weights. Validation: CPU family regression and window fixture, six host executables, native probe syntax, family ownership and formatting checks passed. The new GPU oracle and full trained continuation remain unqualified on this revision. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Build the two changed continuation plans from a pinned original image bundle. Verify its complete digest, original profile, checkpoint, tokenizer and BF16 provenance before loading weights; stream and rehash the six unchanged plans. Record reused plan digests and preserve the original bundle. Share exact runtime metadata with the existing full builder. Runtime and graph math are unchanged. Validate both residency presets, source mismatches, streaming mutation rejection and the existing full build wiring with 59 CPU cases; ownership and formatting checks pass. Full trained execution remains unqualified until the target build and video validation complete. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Allow an explicit second-window prompt on the bounded 226-frame Ref2VA profile. Encode both prompts with the original references, validate child conditioning before first-window inference, and anchor both windows after the longest text prefix as in the fixed upstream implementation. Keep empty and identical prompt behavior, original sampling, media clocks, and residency policies. Add shorter/longer text trajectory cases without changing numerical criteria. Host validation passes; GPU trajectories and trained dialogue quality remain to be verified. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Apply repository-pinned clang-format 22.1.8 to the source-quality diagnostics. Only whitespace and comment wrapping change; preprocessed C++ token streams and all numerical criteria are unchanged. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Recognize checkpoint snapshot identities within the owning family instead of importing the internal provenance collector. Preserve source-bundle and checkpoint matching criteria and compare the local identity against the existing behavior. Keep all direct-staging assertions in the existing bundle-writer test module to preserve the closed shared-tree contract. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Signed-off-by: chaofengw <chaofengw@nvidia.com>
Select one of eight precomposed BF16 video/audio output-head banks in a strongly typed TensorRT graph, retaining explicit operand and output rounding around FP32 accumulation. Add a synthetic GPU oracle for all eight selectors, three dynamic row profiles and the trained hidden width. CPU request validation and scope preflight pass; complete student graph, VSA and trained-model qualification remain required. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Bind both dynamic modalities and the PDD selector before inferring output dimensions in the GPU component oracle. Retain shape diagnostics for each case so a binding failure is distinguishable from numerical disagreement. Keep the original graph, 72 cases and BF16 comparison limits unchanged. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Add segment-pure 128-token transport, independent reference-video key budgets and the learned compression boundary. Implement the operation through TVM-FFI with generic and optional Blackwell backends, and compose it with the existing owner-local H3 backbone and eight PDD output heads. CPU regressions pass 157 cases and seven pinned upstream geometry cases. CUDA compilation, trained numerical validation and native student runtime integration remain required; the earlier output-head mismatch is unwaived. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Select the released eight-forward student recipe explicitly, preserving original H3 scheduling and fixed conditioning. Bind native 128-token VSA geometry and the trained head selector for each window. Validate the closed checkpoint contract, native schedules and transport against pinned source math. Add a real TensorRT VSA component oracle; target GPU compilation and trained-model qualification remain required. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Admit the pinned student contract and exhaustive parameter index, rebuild both student partitions, and reuse only explicitly compatible common plans from a verified source bundle. Write native serialized host buffers directly into the bundle to avoid the second plan-sized Python bytes allocation. Preserve the original bytes API and original continuation defaults. Host checks cover partition ownership, metadata, source rejection, short writes and unchanged common sections; trained model and target memory qualification remain pending. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Distinguish the synthetic text segment from the complete Qwen image-token prefix. Preserve the existing workload and numerical criteria; the component case does not qualify a full six-reference request. Signed-off-by: chaofengw <chaofengw@nvidia.com>
The local block-output ITensor overwrote the caller's serialization stream, so a completed TensorRT build tried to invoke ITensor.write(). Keep the block tensor separate from the optional destination to restore both the default bytes API and native host-buffer streaming. Add four real TensorRT deserialization round trips for bytes/streamed output and features/native-timesteps inputs. CPU validation: 56 passed and six GPU cases deselected. Target GPU execution remains pending; no model shapes, precision boundaries or numerical thresholds change. Signed-off-by: chaofengw <chaofengw@nvidia.com>
The output-head marking loop also reused the serialization destination's name. Keep those ITensors separate so the complete student builder passes the actual writer through to native serialization. This was caught during pre-launch review; no DiT build was submitted with the earlier fix. Validation: 78 CPU checks passed, six GPU cases deselected. All three streaming builder entry points retain their output parameter. Model math, shapes, precision and numerical thresholds remain unchanged. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Allow an explicit denoiser_workspace_bytes budget while retaining the 8-GiB default and the separate AdaLN budget. Record both limits in the build receipt. The full VSA transport graph can require a tactic larger than the previous limit; selecting more scratch does not change its input profiles, weights, precision, sampling or correctness gates. Validation: 96 CPU checks passed and six GPU cases deselected. Coverage includes default and explicit per-plan budgets, invalid values rejected before weight loading, and unchanged student/runtime contracts. Full GPU bundle construction remains pending. Signed-off-by: chaofengw <chaofengw@nvidia.com>
The native Ref2VA pipeline checked every recipe against the original H3 49/98-forward counts after denoising. Valid OmniRef PDD outputs therefore failed before video and audio decoding. Validate the existing recipe contract: original H3 requires 49/98 forwards, and OmniRef PDD requires 8/16. Reject unknown recipes, unsupported window profiles, and all other counts. Preserve the denoising schedule and model precision, shapes, weights, and numerical gates. Add CPU regressions for both recipes and window profiles, including wrong counts and invalid configuration. The focused test reproduces the stale guard failure before this fix; the full pipeline host regression passes after it with both residency modes. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Replay saved operands through the unchanged native FFI function and retain routing, compression, and fine-attention observations. Compare the original gated result against its original reference and numeric limits. A separate zero-gate call exposes fine attention for diagnosis only. It cannot qualify the original failing case or the trained model. Preserve all original input and reference files. Validation: Ruff and Python compilation passed. Real GPU localization remains pending. No production graph or kernel changes are introduced. Signed-off-by: chaofengw <chaofengw@nvidia.com>
The optimized 128-token attention discarded the softmax probability residual before PV. Retain a second BF16 limb in the unused half of the QK tensor-memory tile and accumulate both products into the existing FP32 output. Wait for both limbs before the PV handoff; keep the 64-token path unchanged. A family-owned CPU experiment on saved operands isolates probability storage. Its sampled result supports the change but does not qualify CUDA numerical agreement, performance, trained-model quality or the application. Preserve the existing reference and tolerances for GPU validation. Signed-off-by: chaofengw <chaofengw@nvidia.com>
The released cubic exponential approximation can move BF16 attention across a rounding boundary, which a cancelling gated sum exposes. Use the existing hardware exponential path for the optimized 128-token operation. Preserve the 64-token path, probability residual, workspace and BF16 I/O boundaries. Add a family-owned CPU counterfactual on saved boundaries. It reproduces both fused outputs bitwise and attributes the remaining discrepancies to fine-attention rounding. The cubic simulation explains the observed error pattern; it is diagnostic evidence, not GPU or trained-model qualification. Keep the reference, operands, seeds and numeric limits unchanged. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Accumulate a third BF16 probability limb without increasing the workspace, TMEM or shared-memory allocation. Reuse the epilogue buffer before final PV, wait for the previous TMA read, and hand it back after the MMA completion. Keep the hardware exponential and the original BF16 I/O boundaries. Extend the family CPU diagnostic to project the remaining cancelling queries. Allow the GPU oracle to collect all numeric results before exiting nonzero; its default remains fail-fast and every original comparison still must pass. Reference, comparison, case/seed data and tolerances are unchanged. CUDA numerics, synchronization and performance still require target validation. Signed-off-by: chaofengw <chaofengw@nvidia.com>
The unchanged 56-head component profile includes 708 physical tiles and declares 285624192 bytes of BYOK scratch, exceeding the historical 256 MiB builder pool. Size the oracle builder pool from the declared operation bound while retaining its previous minimum. Keep every operand, reference, case, seed, comparison and tolerance unchanged. This fixes a build-resource error and does not resolve existing numerical failures. The 22 family VSA CPU tests pass. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Add an offline diagnostic for immutable saved failing coordinates. Compare FP64 attention with approximate FP32 scaling and softmax operation orders, retaining direct nearest-even BF16 rounding across FP32 midpoints. Keep native fine values explicitly conditional until directly recorded. The diagnostic does not replace the GPU oracle or change inputs, masks, precision boundaries or tolerances. All 29 focused VSA and rounding CPU tests pass. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Record call geometry and host entry/enqueue boundaries only when the explicit trace environment switch is enabled. Preserve all device computations, original asynchronous behavior and correctness gates. Document that enqueues and optional launch-blocking diagnostic times are not benchmark evidence. Validation: all 29 existing VSA layout, transport and rounding-observation CPU checks pass. CUDA compilation, device-instruction comparison and bounded native failure localization remain separate checks. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Keep the third probability limb in dedicated shared tiles rather than borrowing the output epilogue buffers and waiting on their TMA lifetime. Reassign two KV ring stages to the tiles within the existing shared-memory reservation; retain all three PV products and the external scratch contract. Include absolute host trace timestamps to distinguish a stalled call from setup time. CPU layout and rounding contracts pass; GPU output parity and full-request progress remain to be verified. Signed-off-by: chaofengw <chaofengw@nvidia.com>
A one-slot KV ring cannot prefill both K tiles before Q: the producer waits for MMA to release the first K slot while MMA is waiting for Q. Prefill Q first when the ring cannot hold the initial K loads. Keep the four-slot 64-token order, probability limbs and shared-memory reservation unchanged. CPU layout and rounding contracts pass. The saved-output preservation and original full-request stall checks remain required. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Replace generic online attention with separately scaled FP32 Q/K, explicit masked softmax and FP32 PV inside the owning TVM-FFI binding. Complete compression first and reuse its dead scratch while retaining routing indices, counts and compressed output. Keep optimized Blackwell arithmetic available and reject old generic bundles with insufficient scratch. The existing 56-head large workspace sizes stay unchanged. Host layout checks and 32 focused CPU tests pass; integrated GPU and trained-model qualification remain required. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Singleton SGEMM changes the PV accumulation order on the complete saved interleaved case. Use two independent batched products while retaining one head of residency: dead Q holds one output, the fine tile the other, and the completed key mask holds device pointers. Keep every arena size unchanged and assert output/input/pointer-table separation in the host workspace checks. All 32 focused CPU tests pass; complete integrated GPU gates remain required. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Use the existing FP32 warp and block reductions with explicit division for compression probabilities. Valid INT32 tile sizes directly mask empty keys, retaining the external workspace and BF16 boundaries. A complete saved-output diagnostic isolated the sixteen remaining large-case failures to compression rounding after the old reduction. Keep fine attention and the runtime ABI unchanged. Updated complete-case GPU qualification remains pending. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Keep the compressed output at the front and routing indices at the aligned tail of caller-provided scratch. Fit up to 32 independent original query tiles between them while retaining one head of FP32 K/V, separate regional masks, and disjoint QK/PV destinations. Reuse dead PV output and completed masks for device pointer tables. Preserve the duplicate PV reduction when only one query is active, and leave the external workspace contract and BF16 boundaries unchanged. CPU lifetime and capacity checks pass; complete GPU numeric and performance evidence remain required. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Rewrite the family README around what a user needs: the build-omniref command and its disk and memory needs, the Python entry point, presets, reference capacity, bundle layout, the request contract, continuation, the OmniRef student, plan reuse and the tests. Drop measurements of individual builds and requests, experiment history and the commands of the removed investigation scripts. Keep the known failing trained encoder comparison as an open gap. Signed-off-by: chaofengw <chaofengw@nvidia.com>
apps/short_drama is a small web app that runs every model step through trtmc-server on the user's own GPU: a language model plans the cast, places and shot lists from the novel, Qwen-Image draws reference stills to approve, MiniMax H3 (FastH3 OmniRef) films each shot with sound from the approved stills, and Whisper times the subtitles before FFmpeg assembles 1080x1920 episodes. `python -m trtmc_short_drama up` picks a tier by GPU memory, builds missing bundles with the TRTMC builder (H3 through the family recipe) and loads only the bundles each step needs. The flow (steps, models, parameters) is versioned JSON edited in the app, and a step can be added as a plugin file; dialogue_polish is the example. Planning can use an optional OpenAI-compatible API whose key stays in a mode-600 file. The bundled sample is the public-domain text of 《阿Q正传》 by Lu Xun. Signed-off-by: chaofengw <chaofengw@nvidia.com>
…ds it trtmc-server reads a bundle once into GPU memory. On GB300 the GPU's memory is a NUMA node of the host, and the file cache of a 100 GB bundle can land there as memory no process owns: a later stage's model then fails to load. The engine now advises the kernel to drop the cached pages of the bundles it starts with, before the start and again once the server is ready. On a GB300, the cache an earlier run of the Qwen3.6-27B bundle had left on the GPU's node fell from 52 GB to 0.4 GB once the model had loaded; all GPU memory in use then belonged to the server. Signed-off-by: chaofengw <chaofengw@nvidia.com>
…idea The app now leads a project through its four stages and keeps a person in charge of every step that costs GPU time. - Each stage (plan, references, shots, episodes) shows what is done and what to do next. - Plan checks run on every change: people the chapters name often but nobody plays, someone acting in a shot without being in it, a speaker outside the shot, a line too long for its shot, unknown ids, shots longer than one clip, episodes over their limit. Each line is marked as quoted from the novel or adapted. - Three actions run with the plan step's language model: review (notes per shot on missing events, continuity and over-full shots), revise (rewrite one shot, character, location or prop from a note and keep the rest of the plan) and story (write a short story from a one-line idea, so a project can start without a novel). A plugin step becomes such an action with model_from. - A re-plan, revision, save or restore first keeps the plan it replaces; earlier versions can be restored. - Only characters, locations and props that appear in a shot get a still, and each still lists its shots. - Every take keeps the prompt and stills it was rendered from. A redrawn still or a changed prompt marks the take outdated, and only outdated shots need rendering again. - Shots start with one sample. A batch first says how much GPU time it takes, from the times measured per still and take on this machine. A strip shows every shot of the episode. - A cast member who is heard but not seen, as on a phone, keeps the line: the video prompt gives it as an off-screen voice with that character's voice, instead of moving it to someone in the shot. On a GB300 with Qwen3.6-27B, the review of the bundled four-episode plan took 178 s and found, among others, a line given to the wrong speaker; revising one shot took 12 s. Signed-off-by: chaofengw <chaofengw@nvidia.com>
`up` no longer builds the 27B language model unless asked with `--llm local`. Planning, stories, reviews and revisions use the chat API chosen in Settings, which presets two free tiers (the NVIDIA API catalog and OpenRouter) and still takes any OpenAI-compatible URL. A chat API that answers 429 or 5xx is asked again three times, honoring Retry-After, and a failure carries the API's own reason with what to check. Without any language model the plan stage points to Settings instead of failing on the first step. Signed-off-by: chaofengw <chaofengw@nvidia.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueTip 🎯 Customize your risk assessments
✅ Automatic approvals, with your rules
CodeRabbit Verify
|
The story, plan, still and video prompts named realistic live action themselves, so a plan whose visual style asked for another look, such as cartoon animation, sent contradicting prompts to the image and video models. The plan's "style" is now the only place that names the look; a plan without one keeps the live-action default. The plan prompt offers a cartoon example next to the live-action one, and character looks no longer ask for period clothing. Signed-off-by: chaofengw <chaofengw@nvidia.com>
A model server that exited during its load left the step with only "see <log>", so the page could not say why nothing happened. The error now carries the last line of the log, for example a missing serving module, and the engine reports what it holds, what it is loading and since when, and the end of its log, for the page to show. Signed-off-by: chaofengw <chaofengw@nvidia.com>
…t management - NVIDIA look: NVIDIA Sans, NVIDIA green, light and dark themes, and the header of NVIDIA's documentation sites. - A Projects page with each project's progress, rename and delete. New project asks for the story source, the visual style, the length and a title, and estimates the shots and GPU time. - A project shows five steps and one card with the next thing to do. While a step runs, the card shows its phase (the model load included), the time so far and the usual time; a failed step shows its reason with Try again and the server log. - An activity panel: the loaded models, running and queued steps of every project, recent errors and the end of the trtmc-server log. - A style chosen with the project becomes the plan's style and shapes the looks; shot buttons wait for the pictures they use; the page token is kept across restarts. Signed-off-by: chaofengw <chaofengw@nvidia.com>
…e last one ends TRTMC's MiniMax-H3 starts every clip with a loud burst of about 0.15 s that the official samples do not have. Clips now start 0.25 s in, fade their audio in and out, and the episode gets one loudness. Within an episode, a shot in the same place as the shot before it gets that shot's last frame as an extra reference, so positions, clothes and light carry over. A new take of a shot marks the next one outdated. The shot list asks for one main action per shot, one ambience per scene and no sound effect in the first second. Signed-off-by: chaofengw <chaofengw@nvidia.com>
…ysis and hardware tiers - The page switches between Chinese and English (i18n.js; every page text has a translation, checked by a test). A project has a language: the story, the plan, the lines and the reviews are written in it, and a line is spoken in its own language. Every textarea grows with its text and opens in a large editor. - Video analysis (the analyze action, run with the speech model): measures each rendered shot's opening burst, clipping, level, spoken line, black or frozen frames and the picture and level jump at every cut. With a vision model set in Settings it also sends the frames, up to two reference pictures and the previous shot's last frame, and keeps the model's findings on content, on-screen text, continuity, consistency and reference pictures. The page lists the findings with an editable suggestion and an action; applying them queues plan revisions, new takes or redrawn pictures. - Three hardware tiers by GPU memory: A (140 GB, H3), B (80 GB: RTX PRO 6000, H100, DGX Spark) and C (24 GB), with what each still lacks in TRTMC. GPUs that share system memory (DGX Spark, Jetson Thor) read it from /proc/meminfo. Signed-off-by: chaofengw <chaofengw@nvidia.com>
…hree tiers Also fixes two Chinese strings that repeated 约 and says plainly that an empty vision model means measuring only. Signed-off-by: chaofengw <chaofengw@nvidia.com>
…s missing On an RTX PRO 6000 or a 24 GB card the shots step would only fail later; up now says which TRTMC pieces the tier still needs. --allow-planned starts it anyway for stories, plans and pictures. Signed-off-by: chaofengw <chaofengw@nvidia.com>
…d with the last frame The FastH3 OmniRef student often holds one of the reference pictures still for 0.5 to 1 s and then cuts into the shot (the original 50-step H3 path does not); that held picture made every cut jump. A clip now starts after such a cut when it comes within 1.2 s: nearly still frames, then a change of 40 grey levels or more. Fast motion keeps the 0.25 s lead-in. Each take records where it starts. A shot that continues the shot before it now gets that shot's last frame as Picture 1, and the prompt says the shot opens exactly on it, so a held opening picture becomes a continuation instead of a jump. Signed-off-by: chaofengw <chaofengw@nvidia.com>
… join neighbours only Signed-off-by: chaofengw <chaofengw@nvidia.com>
…ge cannot hear The Whisper bundle is built for Chinese; an English project's analysis now measures the speech but reports no line findings, and says why. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Signed-off-by: chaofengw <chaofengw@nvidia.com>
… unplanned speech Signed-off-by: chaofengw <chaofengw@nvidia.com>
…-off chose Signed-off-by: chaofengw <chaofengw@nvidia.com>
…ames none A flow saved while the app planned through a chat API has no model on its plan step; starting the app with --llm local then failed every story and plan with "model None is not served". Signed-off-by: chaofengw <chaofengw@nvidia.com>
…kes again The FastH3 OmniRef student can hold the previous shot's last frame for 1.4 s, longer than the 1.2 s the opening check looked at, and can hold a portrait and then a prop in turn. A clip now starts after every still picture that ends in a cut within 2.5 s; a picture is still when no frame changes by 3 grey levels or more (held pictures measured at most 2.05, inserts and moving shots at least 4.47 on the clips rendered so far). Assembling an episode cuts takes made under an older rule again from their raw clips and moves their line timing with the new start. Subtitles follow the clips' real lengths, which can now be shorter than planned. The level-jump finding no longer claims that assembly removes the jump. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Assembling ran after the subtitles step, so lines were timed and heard on the old cut. Both steps now cut takes of an older rule again first. A take cut again keeps its moved line timing but drops what was heard on the old cut, and the analysis listens to the new clip. Signed-off-by: chaofengw <chaofengw@nvidia.com>
… no false clipping trtmc-server takes one user message with one picture, so a vision model served by TRTMC rejected the analysis. The shot's frames, its reference pictures and the previous shot's last frame now go as one picture (frames on top, pictures below, described in the prompt), and a request with pictures is one user message that starts with the system text. Thinking models get 12,000 tokens as plans do, and the prompt asks for the project language in its own line: Qwen3.5-9B wrote English findings for a Chinese project before. Every clip was reported as clipping: FFmpeg's mono downmix adds the two channels at 0.71 each, so H3 audio that peaks at 0.95 measured 1.34. The analysis and the speech recognizer's input now take the mean of the channels. Signed-off-by: chaofengw <chaofengw@nvidia.com>
…--vision local) The analysis could only show frames to a vision model of the chat API in Settings, and Settings hold one API for both. Tier A now names Qwen3.5-9B built for pictures as its vision model; with --vision local the app builds and serves it, and the analysis loads it next to speech recognition so the server is not restarted shot by shot. A vision model named in Settings comes first. The local server gets no thinking switch with pictures, which trtmc-server rejects. Signed-off-by: chaofengw <chaofengw@nvidia.com>
…ut tokens TRTMC's Qwen3.5 vision path takes 2,048 tokens of picture and text together, and the analysis of shots with long plans was rejected as invalid. The picture is now 1024 wide with 360-high pictures (about 600 tokens), and each plan field, look and the style are cut to bounded lengths. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Given the recognized speech, Qwen3.5-9B reported fully heard lines as garbled and filed them under on-screen text. Sound and speech are measured already, so the vision prompt drops what was heard, the measurements and the planned sound, and says that only the picture is judged. Signed-off-by: chaofengw <chaofengw@nvidia.com>
Measured on GB300 and RTX PRO 6000: with thinking on (trtmc-server cannot turn it off for pictures) Qwen3.5-9B takes 2 to 4 minutes a shot and ran out of 12,000 tokens on two of three shots of one episode; with thinking off (tried on a patched server copy) it answers in about 35 s but reads the reference pictures as frames of the clip. Signed-off-by: chaofengw <chaofengw@nvidia.com>
H3 had people say lines of its own in shots without one (the analysis heard "我今天要做的" in a shot of the owner's episode). Such shots now say that no one speaks and lips stay closed, as voiceovers already do. Their existing takes show as outdated, since the prompt changed. Signed-off-by: chaofengw <chaofengw@nvidia.com>
…nglish With the plan in the project's language, Chinese prose reached H3 outside the dialogue tags and H3 read it aloud: after the analysis loop revised a shot of the owner's episode into Chinese, the new take spoke the shot summary, and the cartoon project's lines came with extra words. The shots step now asks the language model for the English of every description written in Chinese, Japanese or Korean, before the video model loads, and keeps it per project; prompts, the outdated check and the prompt preview use those translations. Names and lines stay as written. Signed-off-by: chaofengw <chaofengw@nvidia.com>
…the prompt wording A take was marked outdated whenever its saved prompt differed from the prompt the shot would get now. The prompt text mixes the plan with how the app writes prompts, so an app update (the no-one-speaks sentence, descriptions put into English) marked takes outdated that no edit touched. Each take and still now keeps the parts of the plan its prompt was written from. A take is outdated when that plan and the plan now give different prompts, both written the way prompts are written now. Takes rendered before this keep the old check, accepting the descriptions as written and a line-less shot without the no-one-speaks sentence. Signed-off-by: chaofengw <chaofengw@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
TensorRT Model Connect can serve every model a short-drama pipeline needs from one GPU through
trtmc-server: image generation, reference-to-video with audio, and speech recognition. This PR addsapps/short_drama, an end-to-end app that turns a Chinese novel or a one-line idea into vertical 9:16 episodes with sound and subtitles on the user's own GPU.It depends on #1931, which adds MiniMax-H3 reference-to-video and the server's kernel registration and reference video support. Until #1931 merges, this PR also shows that PR's commits; the app itself is the last 28 commits.
Exit Criteria
python -m trtmc_short_drama uppicks the GPU and its tier, builds missing bundles (or takes--bundle NAME=PATH), startstrtmc-serverand serves the web app.--llm localbuilds and servesQwen/Qwen3.6-27Binstead.upnames what TRTMC still lacks for them.Implementation
apps/short_drama/trtmc_short_drama:cli.py:up,--dry-run,--llm api|local,--tier,--vision local(experimental);catalog.py: tiers and models;engine.py: thetrtmc-serverprocess and its calls, with each bundle's page cache dropped once loaded;flow.py: versioned flows and plugin steps;planning.py,prompts.py: the plan step (novel to chapter notes, cast, places, episodes and shot lists) and the prompts for each model;review.py,assist.py: rule checks on every edit, stage progress, and model actions outside the stages (review, one-item revision, a story from an idea);media.py,editing.py,video.py: reference stills, shots (chaining, the English descriptions for H3, the trims above), subtitle checks and FFmpeg assembly;analysis.py: the episode analysis and its suggestions;jobs.py,store.py: one background worker that keeps the GPU on one task at a time, and projects on disk;llm.py: an OpenAI-compatible client that retries 429/5xx withRetry-Afterand reports the API's reason;web.pyandstatic/: a FastAPI app and a single-page UI in Chinese and English (static/i18n.js).samples/ah_q_zheng_zhuan.txt: 《阿Q正传》 by Lu Xun, public domain, recorded inASSET_LICENSES.md.pyproject.toml: addsapps/short_drama/teststo the pytest paths.Change categories
Validation
Commands and Results
CUDA_VISIBLE_DEVICES= python -m pytest apps/short_drama/testson this head: 97 passed.ruff check --config ruff.tomlon every changed Python file andgit diff --check: passed.Hardware, Environment, and Revisions
Head based on
mainat2c2256cc5plus #1931. One NVIDIA GB300 in an aarch64 Ubuntu 24.04 container with TensorRT 11.2.0.113 and CUDA 13.0. Models:Qwen/Qwen-Image,FastVideo/FastVideo-FastH3-Omni-8-Step-V1withMiniMaxAI/MiniMax-H3, andopenai/whisper-large-v3-turbo;Qwen/Qwen3.6-27Bwith--llm local.Not Run / Remaining Gaps
--vision local(Qwen/Qwen3.5-9Bbuilt for pictures) is experimental:trtmc-serverimage chat cannot turn thinking off, so one shot takes minutes and long answers can run out of tokens. A vision model from a chat API is recommended.Contributor Self-Review
Notes For Future Readers
apps/short_drama,ASSET_LICENSES.mdandpyproject.toml.ASSET_LICENSES.mdis a shared file for CI impact, so this PR's Community CI runs the shared smoke families until ci(impact): classify ASSET_LICENSES.md as documentation #1899 lands.apps/short_drama/README.mdcovers requirements, the run options, the language-model presets, use, customizing flows and plugins, and tiers.Risk level
A new, self-contained app directory. Outside it, only the license list and the pytest paths change.